Learning Python’s Decorators and Scope by Touch

#Python

#初心者向け

春田拓海 | Takumi

2019.11.25

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

This post is English translation of the Japanese version. Since I'm not a native English speaker, please let me know if you find sentences which don't make sense. Cheers!

To have a good command of Python's decorators, it is necessary to know what Python's scope is. But when you try to adapt the decorators to your codes, you might realize you don't know how to actually use them if you just understand the concept of the scope.

In this post, we discuss the characteristics of Python's decorators and scope by touch, or without abstract terms. We will take a trial-and-error approach to understand them with a sample code.

var_a ~ var_e: 5 variables
var_f ~ var_h: 3 parameters
outer(var_f): The function nesting inner(var_g)
inner(var_g): The main function
decorator(var_h): The function wrapping inner(var_g)

# Python 3.7.3

var_a = "var_a"

print('--- L3: Define decorator()')
def decorator(var_h):
    var_c = "var_c"
    print('--- L6: In decorator() -> VAR_X = {}'.format(VAR_X))
    
    def _decorator(f):
        var_d = "var_d"
        print('--- L10: In _decorator() -> VAR_X = {}'.format(VAR_X))
        
        def wrapper(var_g):
            print("--- L13: Before decorate")
            print('--- L14: In wrapper() -> VAR_X = {}'.format(VAR_X))
            var_e = "var_e"
            f(var_g)
            print("--- L17: After decorate")
        return wrapper
    return _decorator


print('--- L22: Define outer()')
def outer(var_f):
    print("--- L24: Into outer()")
    var_b = "var_b"
    print('--- L26: In outer() -> VAR_X = {}'.format(VAR_X))

    @decorator("var_h")
    def inner(var_g):
        # Examine which variables we can refer or update within the function "inner"
        print("--- L31: Into inner()")
        print('[Ref] L32: inner() -> VAR_X = {}'.format(VAR_X))
        VAR_X = "CHANGED"
        print('[Chg] L34: inner() -> VAR_X = {}'.format(VAR_X))

    print('--- L36: Execute inner()')
    inner("var_g")
    print('--- L38: In outer()  -> VAR_X = {}'.format(VAR_X))


print('--- L41: Execute outer()')
outer("var_f")
print('--- Finish')

We are going to replace the variable VAR_X by variables from var_a to var_h and modify code to refer or update to the variable from inner(). The goal is to know which variables can be referred to from each scope. In the nested function inner(), one of these variables is referred, updated and referred again. This sample code is also available on my GitHub. You can also try it yourself.

GitHub - TakumiHaruta/decorator_practice: Easy examples for decorators and scopes

Let's get started with var_a.

var_a

var_a is a global variable defined at the top of the code. Once you replace VAR_X by var_a and execute it, you will get the following error.

# Error
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_a = var_a
--- L6: In decorator() -> var_a = var_a
--- L10: In _decorator() -> var_a = var_a
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_a = var_a
--- L31: Into inner()
Traceback (most recent call last):
  File "a_deco.py", line 42, in <module>
    outer("var_f")
  File "a_deco.py", line 37, in outer
    inner("var_g")
  File "a_deco.py", line 16, in wrapper
    f(var_g)
  File "a_deco.py", line 32, in inner
    print('[Ref] L32: inner() -> var_a = {}'.format(var_a))
UnboundLocalError: local variable 'var_a' referenced before assignment

You can see the order of execution in the log above. The function defined first was decorator() at line 3, and outer() at line 22 followed. After outer() at line 41 was executed, inner() was defined with executing decorator(). It seems like that the global variable var_a can be referred to from outer(), decorator() and _decorator(). Then the execution went into inner() at line 36 with executing the inside of wrapper(). Lastly, UnboundLocalError occurred when var_a was referred in inner().

UnboundLocalError means that two var_a, the local one and the global one have the same name but these are different. The print function at line 32 in inner() tries to refer a local var_a first. If it does not exist, the print function is going to search the outer scope of inner() next. You might think "Call the global var_a at line 32" and then "inserted CHANGED into the local var_a at line 34", but this would not work as a human thinks.

There are three ways to refer or update var_a from inner().

#1. Refer `var_a` as the global variable

-> Comment out var_a = "CHANGED" at line 33.

By not defining the local var_a in inner(), all var_a used in inner() are going to search outer scope.

# Result 1
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_a = var_a
--- L6: In decorator() -> var_a = var_a
--- L10: In _decorator() -> var_a = var_a
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_a = var_a
--- L31: Into inner()
[Ref] L32: inner() -> var_a = var_a
[Chg] L34: inner() -> var_a = var_a
--- L17: After decorate
--- L38: In outer()  -> var_a = var_a
--- Finish

#2. Refer and update `var_a` as the local variable

-> Replace the argument of def inner(var_g) at line 29 and the parameter of inner(var_g) at line 37 to var_a.

By passing the global var_a to the parameter of inner(), you can refer the global var_a inside inner(). The value of var_a was updated to CHANGED in inner(), but it was not changed to CHANGED outside of inner() after the execution of inner had been done. The var_a which was assigned the value CHANGED is the local variable of inner(), so the global var_a was not affected by this assignment.

# Result 2
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_a = var_a
--- L6: In decorator() -> var_a = var_a
--- L10: In _decorator() -> var_a = var_a
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_a = var_a
--- L31: Into inner()
[Ref] L32: inner() -> var_a = var_a
[Chg] L34: inner() -> var_a = CHANGED
--- L17: After decorate
--- L38: In outer()  -> var_a = var_a
--- Finish

#3. Refer and update `var_a` as the global variable

-> Use global keyword before referring var_a at line 32; global var_a

By defining var_a as the global keyword inside the inner() function, you can refer and update the global var_a from inner().

# Result 3
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_a = var_a
--- L6: In decorator() -> var_a = var_a
--- L10: In _decorator() -> var_a = var_a
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_a = var_a
--- L31: Into inner()
[Ref] L32: inner() -> var_a = var_a
[Chg] L34: inner() -> var_a = CHANGED
--- L17: After decorate
--- L38: In outer()  -> var_a = CHANGED
--- Finish

var_b

Next, we are going to deal with var_b which is the local variable inside the outer() function.

## Error 1
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_b = var_b
Traceback (most recent call last):
  File "b_deco.py", line 42, in <module>
    outer("var_h")
  File "b_deco.py", line 28, in outer
    @decorator("var_g")
  File "b_deco.py", line 6, in decorator
    print('--- L6: In decorator() -> var_b = {}'.format(var_b))
NameError: name 'var_b' is not defined

Unlike var_a, var_b can only be referred inside outer() and functions in decorator() cannot reach it out. There are several ways to refer var_b from decorator(), but this time we are going to just comment out the print functions in decorator().

## Error 2
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_b = var_b
--- L36: Execute inner()
--- L13: Before decorate
--- L31: Into inner()
Traceback (most recent call last):
  File "b_deco.py", line 42, in <module>
    outer("var_f")
  File "b_deco.py", line 37, in outer
    inner("var_g")
  File "b_deco.py", line 16, in wrapper
    f(var_g)
  File "b_deco.py", line 32, in inner
    print('[Ref] L32: inner() -> var_b = {}'.format(var_b))
UnboundLocalError: local variable 'var_b' referenced before assignment

After going into inner(), UnboundLocalError occurred at line 32. The ways of referring to var_b are exactly the same as #1 and #2 of var_a, i.e., comment out var_b = "CHANGED" at line 33 or pass var_a to the parameter of inner(). The way of referring to and updating var_b like #3 is to use nonlocal keyword instead of global keyword.

The result of adding nonlocal var_b was the following. The change of var_b in inner() was reflected on the one in outer() at line 38.

## Result
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_b = var_b
--- L36: Execute inner()
--- L13: Before decorate
--- L31: Into inner()
[Ref] L32: inner() -> var_b = var_b
[Chg] L34: inner() -> var_b = CHANGED
--- L17: After decorate
--- L38: In outer()  -> var_b = CHANGED
--- Finish

var_c & var_d & var_e

All variables created in decorator(); var_c, var_d and var_e are defined in different scopes. But in the case of just referring from inner(), the difference between these scopes doesn't matter. Let's only take var_c as a representative.

First of all, var_c cannot be called from outer() because decorator() has the different scope from the one of outer(). We should comment out line 26 and 38 before executing the code.

## Error
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L6: In decorator() -> var_c = var_c
--- L10: In _decorator() -> var_c = var_c
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_c = var_c
--- L31: Into inner()
Traceback (most recent call last):
  File "c_deco.py", line 42, in <module>
    outer("var_f")
  File "c_deco.py", line 37, in outer
    inner("var_g")
  File "c_deco.py", line 16, in wrapper
    f(var_g)
  File "c_deco.py", line 32, in inner
    print('[Ref] L32: inner() -> var_c = {}'.format(var_c))
UnboundLocalError: local variable 'var_c' referenced before assignment

We caught UnboundLocalError. var_c cannot be directly referred to from inner(), so to comment out var_c = "CHANGED" like the example of #1 would not work. In order to refer to var_c from inner(), we need to pass var_c to the parameter of inner() like #2. The result of changing to f(var_c) and def inner(var_c) at line 16 and 29 is the following.

# Result
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L6: In decorator() -> var_c = var_c
--- L10: In _decorator() -> var_c = var_c
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_c = var_c
--- L31: Into inner()
[Ref] L32: inner() -> var_c = var_c
[Chg] L34: inner() -> var_c = var_c
--- L17: After decorate
--- Finish

Note that the function wrapped by the decorator cannot directly refer objects in the scope of the decorator itself. Functions in decorator(), _dacorator() and wrapper() can directly refer to var_c, but outer() and inner() which have different scopes cannot refer to it, and vice versa. We can see the word 'wrapper' when using decorators, but the word 'background' would be better to explain a decorator than it. The word 'wrapper' might somehow be misleading.

var_f

Next, we are going to refer the objects made of arguments, not variables.

var_f is the arguments object of outer(). It cannot be called from decorator(), so we need to comment out line 6, 10 and 14 before execution.

# Error
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L26: In outer() -> var_f = var_f
--- L36: Execute inner()
--- L13: Before decorate
--- L31: Into inner()
Traceback (most recent call last):
  File "f_deco.py", line 42, in <module>
    outer("var_f")
  File "f_deco.py", line 37, in outer
    inner("var_g")
  File "f_deco.py", line 16, in wrapper
    f(var_g)
  File "f_deco.py", line 32, in inner
    print('[Ref] L32: inner() -> var_f = {}'.format(var_f))
UnboundLocalError: local variable 'var_f' referenced before assignment

This case is almost the same as var_b in outer(). You can refer it as the ways of #1 or #2, or by using nonlocal keyword for var_b. Note that nonlocal keyword is valid for objects made of arguments.

var_g

var_g is the arguments object of inner(). It is created when inner() is called, and not able to be referred from outer() and decorator(). Comment out line 6, 10, 14, 26 and 38.

# Result
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L36: Execute inner()
--- L13: Before decorate
--- L31: Into inner()
[Ref] L32: inner() -> var_g = var_g
[Chg] L34: inner() -> var_g = CHANGED
--- L17: After decorate
--- Finish

There was not any error in the case of var_g.

var_h

Finally, we are going to take var_h which is the argument object of decorator(). Comment out line 6, 10 and 14 before execution because they cannot be referred to from decorator().

# Error
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L6: In decorator() -> var_h = var_h
--- L10: In _decorator() -> var_h = var_h
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_h = var_h
--- L31: Into inner()
Traceback (most recent call last):
  File "h_deco.py", line 42, in <module>
    outer("var_f")
  File "h_deco.py", line 37, in outer
    inner("var_g")
  File "h_deco.py", line 16, in wrapper
    f(var_g)
  File "h_deco.py", line 32, in inner
    print('[Ref] L32: inner() -> var_h = {}'.format(var_h))
UnboundLocalError: local variable 'var_h' referenced before assignment

This case is also same as the result of var_c. You can fix the error by passing var_h to the argument of inner() in wrapper().

# Result
--- L3: Define decorator()
--- L22: Define outer()
--- L41: Execute outer()
--- L24: Into outer()
--- L6: In decorator() -> var_h = var_h
--- L10: In _decorator() -> var_h = var_h
--- L36: Execute inner()
--- L13: Before decorate
--- L14: In wrapper() -> var_h = var_h
--- L31: Into inner()
[Ref] L32: inner() -> var_h = var_h
[Chg] L34: inner() -> var_h = CHANGED
--- L17: After decorate
--- Finish

Now, you might realize that the one of attributes of Python's decorator is that you can alter arguments of the function wrapped in the decorator. The decorator can handle the same arguments as the ones of the function wrapped by it. On the other hand, the objects used in wrapper() cannot be referred to from other scopes unless the objects are passed to the parameter of wrapper().

Appendix: Examining a complex decorator

In addition to the above experiments, we will try to replace all variables and parameters to var_i and update them at each scope. Are you wondering what value each var_i has?

The code is the following. I added nonlocal var_i to the line 8 because we cannot update the objects in decorator() from _decorator() directly.

print('--- L1: Define decorator()')
def decorator(var_i):
    print('--- L3: In decorator() -> var_i = {}'.format(var_i))
    var_i = "decorator"
    print('--- L5: In decorator() -> var_i = {}'.format(var_i))
    
    def _decorator(f):
        nonlocal var_i
        print('--- L9: In _decorator() -> var_i = {}'.format(var_i))
        var_i = "_decorator"
        print('--- L11: In _decorator() -> var_i = {}'.format(var_i))
        
        def wrapper(var_i):
            print("--- L14: Before decorate")
            print('--- L15: In wrapper() -> var_i = {}'.format(var_i))
            var_i = "wrapper"
            f(var_i)
            print("--- L18: After decorate")
        return wrapper
    return _decorator


print('--- L23: Define outer()')
def outer():
    print("--- L25: Into outer()")
    var_i = "var_i"
    print('--- L27: In outer() -> var_i = {}'.format(var_i))

    @decorator(var_i)
    def inner(var_i):
        # Examine which variables we can refer or update within the function "inner"
        print("--- L32: Into inner()")
        print('[Ref] L33: inner() -> var_i = {}'.format(var_i))
        var_i = "CHANGED"
        print('[Chg] L35: inner() -> var_i = {}'.format(var_i))

    print('--- L37: Execute inner()')
    inner(var_i)
    print('--- L39: In outer()  -> var_i = {}'.format(var_i))


print('--- L42: Execute outer()')
outer()
print('--- Finish')

## Result
--- L1: Define decorator()
--- L23: Define outer()
--- L42: Execute outer()
--- L25: Into outer()
--- L27: In outer() -> var_i = var_i
--- L3: In decorator() -> var_i = var_i
--- L5: In decorator() -> var_i = decorator
--- L9: In _decorator() -> var_i = decorator
--- L11: In _decorator() -> var_i = _decorator
--- L37: Execute inner()
--- L14: Before decorate
--- L15: In wrapper() -> var_i = var_i
--- L32: Into inner()
[Ref] L33: inner() -> var_i = wrapper
[Chg] L35: inner() -> var_i = CHANGED
--- L18: After decorate
--- L39: In outer()  -> var_i = var_i
--- Finish

After defining decorator() and outer(), var_i has the value var_i until it is replaced to the value decorator at line 4. But after it exits decorator(), the value of var_i is restored to var_i. When inner() is executed, var_i has the value wrapper which is assigned in wrapper(). What the complicated feature it is!

Okay, let's wrap up these examinations.

Summary

Be aware of scope relationships to define decorators effectively

If you would ignore scope relationships, you might not be able to implement decorators well. Make sure there are no dependencies between objects in a decorator and ones in a wrapped function. Also, keep in mind to make decorators general-purpose for various functions.

You can alter arguments of a wrapped function within a decorator

One of the powerful features of Python's decorator is brought out when performing a common process for arguments. The web application framework, Flask is a superb example of using decorators.

View Decorators — Flask Documentation (1.1.x)

Define a wrapped function AFTER executing outside of `wrapper()`

As you can see the examine of var_i, the outside of wrapper() is executed when the wrapped function is defined, while the inside of wrapper() is executed when the wrapped function is called. You need to understand the difference between them and decide which process is supposed to be written inside or outside wrapper().

`global` and `nonlocal` does not mean altering local variables to global variables and nonlocal variables

It was just my misunderstanding, but global keyword and local keyword do not mean that they redefine local variables as global and nonlocal. The behavior of these keyword is tying a local variable to an outer variable with the same name. If a global variable with the same name as a local variable doesn't exist, you cannot use global keyword for it.